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ABSTRACT 

For 15 years, the Software Engineering Laboratory (SEL) ha 5 been 
carrying out studies and experiments for the purpose of understand- 
ing, assessing, and improving software and software processes 
within a production software development environment at the Na- 
tional Aeronautics and Space Administration/Goddard Space Flight 
Center (NASA/GSFQ. The SEL comprise! three major organiza- 
tions: 

• NASA/GSFC, Flight Dynamics Division 

• University of Maryland, Department of Computer Sci- 
ence 

• Computer Sciences Corporation, Flight Dynamics 
Technology Group 

These organizations have jointly carried out several hundred 
software studies, producing hundreds of reports, papers, and 
documents, all of which describe some aspect of the software en- 
gineering technology that has been analyzed in the flight dy- 
namics environment at NASA. The studies range from small, 
controlled experiments (such as analyzing the effectiveness of 
code reading versus that of functional testing) to large, multiple- 
project studies (such as assessing the impacts of Ada on a pro- 
duction environment). The organization*! driving goal is to im- 
prove the software process continually, so that sustained 
improvement may be observed in the resulting products. This 
paper discusses the SEL as a functioning example of an opera- 
tional software experience factory and summarizes the charac- 
teristics of and major lessons learned from 15 years of SEL 
operations. 

1. THE EXPERIENCE FACTORY CONCEPT 
Software engineering has produced a fair amount of research and 
technology transfer in the first 24 years of its existence. People 
have built technologies, methods, and tools that are used by many 
organizations in development and maintenance of software 
systems. 

Unlike other disciplines, however, very little research has been 
done in the development of models for the various components of 
the discipline. Models have been developed primarily for the 
software product, providing mathematical models of its function 
and structure (e.g., finite state machines in object-oriented design), 
or, in some advanced instances, of its observable quality (e.g., reli- 
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ability models). However, there has been very little modeling of 
several other important components of the software engineering 
discipline, such as processes, resources, and defects. Nor has much 
been done toward understanding the logical and physical in- 
tegration of software engineering models, analyzing and evaluating 
them via experimentation and simulation, and refining and tailoring 
them to the characteristics and the needs of a specific application 
environment 

Currently, research and technology transfer in software engineering 
are donemostly bottom-up andin isolation. Topfovide software engi- 
neering with a rigorous, scientific foundation and a pragmatic frame- 
work, the following are needed [1]: 

• A top-down, experimental, evolutionary framework in 
which research can be focused and logically integrated to 
produce models of the discipline, which can then be 
evaluated and tailored to the application environment 

• An experimental laboratory associated with the software 
artifact that is being produced and studied to develop and 
refine comprehensive models based upon measurement 
and evaluation 

The three major concepts supporting this vision are 

• A concept of evolution: die Quality Improvement Para- 
digm [2] 

• A concept of measurement and control: the Goal/ 
Qoestion/Metric Approach [3] 

• A concept of the organization: the Experience Factory 

[4] 

The Quality Improvement Paradigm is a two-feedback loop 
process (project and organization loops) that is a variation of the 
scientific method. It coosists of the following steps: 

• Characterization: Understand the environment based 
upon available models, data, intuition, etc., so that simi- 
larities to other projects can be recognized 

• Planning: Based on this characterization: 

- Set quantifiable goals for successful project and or- 
ganization p erf or m ance and improvement 

- Choose the app ro pri ate processes far improvement, 
and s u pport in g methods and tools to achieve the goals 
in the given environment 

• Execution: Pe rform the processes while constructing the 
products and provide real-time project feedback based on 
the goal achievement data 

• Packaging: At the end of each specific project: 

— Analyze the data and the information gathered to 
evaluate the current practices, determine problems. 

Hi WD 


2-3 


10006788L 



record findings, and make recommendations for 
future project improvements 

- Package the experience gained in the form of updated 
and refined models and other forms of structured 
knowledge gained from this and prior projects 

- Store the packages in an experience base so they are 
available for future projects 

The Goal/Question/Metric Approach is used to define measure* 
ment on the software project, process, and product in such a way that 

• Resulting metrics are tailored to the organization and its 
goals 

• Resulting measurement data play a constructive and 
instructive role in the organization 

• Metrics and their interpretation reflect the quality values 
and the different viewpoints (developers, users, opera- 
tors, etc.) 

Although originally used to define and evaluate a particular project 
in a particular environment, the Goal/Question/Metric Approach 
can be used for control and improvement of a software project in 
the context of several projects within the organization [5,6]. 

The Goal/Questioo/Metric Approach defines a measurement model 
on three levels: 

• Conceptual level (goal): A goal is defined for an object, 
for a variety of reasons, with respect to various models of 
quality, from various points of view, and relative to a par- 
ticular environment 

• Operational level (question): A set of questions is used 
to define models of the object of study and the focuses 
on that object to characterize the assessment or achieve- 
ment of a specific goal 

• Quantitative level (metric): Asetofmetrics.basedonthe 
models, is associated with every question in order to an- 
swer it in a quantitative way 

The concept of the Experience Factory was introduced to institu- 
tionalize tite collective learning of the organization that is at the 
root of continual improvement and competitive advantage. 

Reuse of experience and collective learning cannot be left to the 
imagination of individual, very talented, managers: they become a 
corporate concern, like the portfolio of a business or company 
assets. The experience factory is the organization that supports 
reuse of experience and collective learning by developing, updat- 
ing, and delivering, upon request to the project organizations, dus- 
ters of competencies that the SEL refers to as experience packages. 
The project organizations offer to the experience factory their 
products, the plans used in their development, and the data gath- 
ered during development and operation (Figure 1). The experience 
factory transforms these objects into reusable units and supplies 
them to the project organizations, together with specific support 
that includes monitoring and consulting (Figure 2). 

The experience factory can be a logical and/or physical organization, 
but it is important that its activities are separated and made inde- 
pendent from those of the project organization. The packaging of 
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Figure 2. Experience Factory Functions 


experience isbased on tenets and techniques that are different from tl 
problem solving activity used in project development [7]* 

On the one hand, from the perspective of an organization producii 
software, the difference is outlined in the following chart: 


PROJECT ORGANIZATION 
(Problem Solving) 

EXPERIENCE FACTORY 
(Experience Packaging) 

Decomposition of a problem into 
simpler ones 

Unification of different solutions 
and redefinition of the problem 

Instantiation 

Generalization, formalization 

Derigftfmplementatioo process 

Analysis/synthesis process 

Validation and verification 

Experimentation 
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On the other hand, from the perspective of software engineering 
research, there are the following goals: 


PROJECT ORGANIZATION 
(Problem Solving) 

EXPERIENCE FACTORY 
(Experience Packaging) 

Develop representative languages for 
products 
processes 

Develop techniques for 
abstraction 
generalization 
tailoring 
formalization 
analysis/synthesis 

Develop techniques for 
desigiVimplementation 

data collectiof^validatjoci/ 

analysis 

validation and verification 

Experiment with techniques 

Build automatic support tools 

Package and integrate for reuse 
experimental results 
processes/products 


In a correct implementation of the experience factory paradigm, the 
projects and the factory will have different process models. Each 
project will choose its process model based on the characteristics of 
the software product that will be delivered, whereas the factory will 
define (and change) its process model based upon organizational 
and performance issues. The main product of the experience fac- 
tory is the experience package. There are a variety of software 
engineering experiences that can be packaged: resource baselines 
and models; change and defect baselines and models; product 
baselines and models; process definitions and models; method and 
technique models and evaluations; products; lessons learned; and 
quality models. The content and structure of an experience pack- 
age vary based on the kind of experience clustered in the package. 
There is, generally, a central element that determines what the pack- 
age is: a software life-cycle product or process, a mathematical 
relationship, an empirical or theoretical model, a data base, etc. 
This central element can be used to identify the experience package 
and produce a taxonomy of experience packages based on the 
characteristics of this central element 

• Product packages (programs, architectures, designs) 

• Tool packages (constructive and analytic tools) 

• Process packages (process models, methods) 

• Relationship packages (cost and defect models, resource 
models, etc.) 

• Management packages (guidelines, decision support 
models) 

• Data packages (defined and validated data, standardized 
data, etc.) 

The structure and functions of an efficient implementation of the 
experience factory concept are modeled on the characteristics and 
the goals of die organization it supports. Therefore, different levels 
of abstraction best describe the architecture of an experience factory 
in order to introduce the specificity of each environment at the right 
level without losing die rep rese ntation of the global picture and die 
ability to compare different solutions [8]. 

The levels of abstraction that the SELproposes to represent the archi- 
tecture of an experience factory are as follows: 

• Reference level: This first and more abstract level rep- 
resents the activities in the experience factory by 
active objects, called architectural agents. They are 


specified by their ability to perform specific tasks and 
to interact with each other. 

• Conceptual level: This level represents the interface of 
the architectural agents and the flows of data and control 
among them. They specify who communicates with 
whom, what is done in the experience factory, and what 
is done in the project organization. The boundary of the 
experience factory, Le., the line that separates it from the 
project organization, is defined at this level based on the 
needs and characteristics of an organization. It can 
evolve as these needs and characteristics evolve. 

• Implementation level: This level defines the actual 
technical and organizational implementation of the ar- 
chitectural agents and their connections at the conceptual 
level. They are assigned process and product models, 
synchronization and communication rules, and appropri- 
ate perform er s (people or computers). Other implementa- 
tion details, such as mapping the agents over organiza- 
tional departments, are included in the specifications 
provided at this leveL 

The architecture of the experience factory can be regarded as a spe- 
cial instance of an experience package whose design and evolution 
are based on the levels of abstraction just introduced and on the 
methodological framework of the improvement paradigm applied 
to the specific architecture. 

The Software Engineering Laboratory (SEL) is an operating ex- 
ample of an e xperi ence factory. Figure 3 shows the conceptual 
level of the SEL experience factory, identifying the primary archi- 
tectural agents and the interactions among them. The remaining 
sections describe the SEL implementation of the experience factory 
concept They discuss its background, operations, and achieve- 
ments, and assess the impact it has had on the production environ- 
ment it supports. 



Figure 3. The SEL— Conceptual Level 


2. SEL BACKGROUND 

The SEL was established in 1976 as a cooperative effort among the 
University of Maryland, the National Aeronautics and Space 
Administratioo/Goddard Space Flight Center (NASA/GSFC), and 
Computer Sciences Corporation (CSC). Its goal was to understand 
and improve the software development process and its products 
within GSFCs Flight Dynamics Division (FDD). At that time, al- 
though significant advances were being made in developing new 
technologies (e.g., structured development practices, automated 
tools, quality assurance approaches, and management tools), there 
was very limited empirical evidence or guidance for applying these 
promising, yet immature, techniques. Additionally, it was apparent 
that there was very limited evidence available to qualify or to 
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quantify the existing software process and associated products, let 
alone understand the impact of specific process methods. Thus, the 
SEL staff initiated efforts to develop some means by which the 
software process could be understood (through measurement), 
qualified, and measurably improved through continually expanding 
understanding, experimentation, and process refinement. 

This working relationship has been maintained continually since its 
inception with rclati vely little change to the overall goals of the orga- 
nization. In general, these goals have matured rather than changed; 
they axe as follows: 

1 . Understand: Improve insight into the software process and 
its products by characterizing a production environment 

2. Assess: Measure the impact that available technologies 
have on the software process. Determine which technolo- 
gies are beneficial to the environment and, most important- 
ly, how the technologies most be refined to best match the 
process with the environment 

3. Package/Infuse: After identifying process improvements, 
package the technology in aform that allows it to be applied 
in the production organization. 

These goals are addressed sequentially, in an iterative fashion, as 
shown in Figure 4. 



Figure 4 SEX Process Improvement Steps 


The approach taken to attaining these goals has been to apply 
potentially beneficial techniques to the development of production 
software and to measure the process and product in enough detail 
to quantWably assess the applied technology. Measures of con- 
cern, such as cost, reliability, and/or maintainability, are defined as 
the organization determines the major near- and long-tom objec- 
tives for its software development process improvement program. 
Once those objectives are known, the SEL staff designs the experi- 
ment; that is, it defines the particular data to be captured and the 
questions that must be addressed in each experimental project 

All of the experiments conducted by the SEL have occurred within 
the production environment of the flight dynamics software devel- 
opment facility at NASA/GSFC. The SEL production environ- 
ment consists of projects that are classified as mid-sized software 
systems. The average project lasts 2 to 3- 1/2 years, with an average 
staff size of 15 software developers. The average software size is 
175 thousand source lines of code (KSLOC), counting commen- 
tary, with about 25 percent reused from previous development 


efforts. Virtually all projects in this environment are scientific 
ground-based systems, although some embedded systems have 
been developed. Most software is developed in FORTRAN, al- 
though Ada is starting to be used more frequently. Other lan- 
guages, such as Pascal and Assembly, are used occasionally. Since 
this environment is relatively consistent, it is conducive to the 
experimentation process. In the SEL, there exists a homogeneous 
class of software, a stable development environment, and a con- 
trolled, consistent, management and development process. 

3. SEL OPERATIONS 

The following three major functional groups support the exper- 
imentation and studies within the SEL (Figure 5): 

1. Software developers, who are responsible for producing 
the flight dynamics application software 

2. Software engineering analysts, who are the researchers 
responsible for carrying out the experimentation process 
and producing study results 

3. Data base support staff, who are responsible for collect- 
ing, checking, and archiving all of fire information col- 
lected from the development efforts 

During the past 15 years, the SEL has collected and archived data 
on over 100 software development projects in the organization. 
The data are also used to build typical project profiles against 
which ongoing projects can be compared and evaluated. The SEL 
provides managers in this environment with tools (online and 
paper) for monitoring and assessing project status. 

Typically, there are 6 to 10 projects simultaneously in progress in 
die flight dynamics environment As was mentioned earlier, they 
average 175 KSLOC, ranging from small (6-8 KSLOC) to large 
(300- 400 KSLOC), with a few exceeding 1 million source lines of 
code (MSLOQ. Each project is considered an experiment within 
the SEL, and the goal is to extract detailed information to un- 
derstand the process better and to provide guidance to future 
projects. 

To sup por t the studies and to support the goal of continually 
increasing understanding of the software development process, the 
SEL regularly collects detailed data from its development projects. 
The types of data collected include cost (measured in effort), 
process, and product data. Process data include information about 
the project, such as the methodology, tools and techniques used, 
and information about personnel experience and training. Product 
data include size (in SLOC), change and error information, and the 
results of postdevelopment static analysis of the delivered code. 

The data may be somewhat different from one project to anothei 
since the goals for a particular experiment may be different between 
projects. There is a basic set of information (such as effort aiW 
error data) that is collected for every project However, as change* 
are made to specific processes (e.g., Ada projects), the detailed dafc 
collected may be modified. For example, figure 6 shows th< 
standard error report form, used on all projects, and the modifiet 
Ada version, used for specific projects where Ada is being studied 

As the information is collected, it is quality assured and placed in ; 
central data base. The analysts then use these data together wid 
other information, such as subjective lessons learned, to analyze tb 
impact of a specific software process and to measure and then fee* 
back results to both ongoing projects and follow-on projects. 

The' data are used to build predictive models for future projects an 
to provide a rationale for refining particular software process* 
being used. As the data are analyzed, papers and reports are genei 
ated that reflect results of the numerous studies. Additionally, tb 
results of the analysis are packaged as standards, policies, trainin 
materials, and management tools. 
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Figure 5. SEL Functional Groups 


4. SEL DATA ANALYSIS 

The overall concept of the e xpe rie nce factory h as continually 
matured within the SEL as understanding of the software process 
has increased. The experience factory goal is to demonstrate 
continual improvement of the software process within an environ- 
ment by carrying out analysis, measurement, and feedb a ck to 
projects within the environment. The steps, previously described, 
include understanding, assessment/refinement, and packaging. 
The data described in the previous section are uaed as one major 
element that supports these three activities in the SEL. lathis sec- 
tion, examples are given to demonstrate the major stages of the 
experience factory. 

4.1. UNDERSTANDING 

Understanding what an organization does and bow dial orga- 
nization operates is fundamental to any attempt to plan, manage, or 
improve the software process. This is especially true for software 
development organizations. The following two examples illustrate 
how understanding is supported in an operation such as die SEL. 

Effort distribution (Le., which phases of the life cycle consume 
what portion of development effort) is one baseline characteristic of 
the SEL software development process. Figure 7 presents the effort 
distributions for 11 FORTRAN projects, by life-cycle phase and by 
activity. The phase data count hours dunged to a project during 
each calendar phase. The activity data count all hours attributed to 
a particular activity (as reported by the programmer), regardless of 
when in die life cycle the activity occurred. Understanding these 
distributions is important to assessing the similarities/differences 
observed on an ongoing project, planning new efforts, and evaluat- 
ing new technology. 


The er ror detection rate is another interesting model from the SEL 
environment There are two types of information in this model. 
The first is the absolute error rate expected in each phase. By 
collecting the information on software errors, the SEL has 
constructed a model of the expected error rate in each phase of the 
fife cycle. The SEL expects about four errors per 1000 SLOC dur- 
ing implementation: two during system test, one during acceptance 
test, and one-half during operation and maintenance. Analysis of 
more recent projects indicates that these absolute error rates are de- 
clining as the software development process and technology 
improve. 

The trend that can be derived from this mode] is that the error 
detection rates reduce by 50 percent in each subsequent phase 
(Figure 8). This pattern seems to be independent of the actual 
values of the error rates; it is still true in the recent projects where 
the overall error rates are declining. This model of error rates, ms 
well as numerous other similar types of models, can be used to 
better predict, manage, and assess change on newly developed 
projects. 

4*2. ASSESSING/REFINING 

In the second major stage of the experience factory, elements of the 
process (such as specific software development techniques) are as- 
sessed, and die evolving technologies are tailored to the particular 
environment Each project in the SEL is considered to be an ex- 
periment in which some software method is studied in detail. 
Generally, the subject of the study is a specific modification to the 
standard process, a process that obviously comprises numerous 
software methods. 
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Figure 8. Derived SEL Error Model 

One recent study that exemplifies the assessment stage involves the 
Cleanroom software methodology [9]. This methodology has been 
applied on three projects within the SEL, each providing additional 
insight into the Qeanroom process and each adding some element 
of “refinement” to the methodology for this one environment 

The SEL trained teams in the methodology, then defined a 
modified set of Qeanroom-specific data to be collected. The 
projects were studied in an attempt to assess the impact that Clean- 
room had on the process as well as on such measures as 
productivity and reliability. Figure 9 depicts the characteristics of 
the Qeanroom changes, as well as the results of the three experi- 
ments. 

The Qeanroom experiments included significant changes to the 
standard SEL development methodology, thereby requiring ex- 
tensive training, p re pa ration, and careful execution of the studies. 
Detailed experimentation plans were generated for each of the 
studies (as they are for all such e xp e rim ents), and each included a 
description of the goals, die questions that had to be addressed, and 
the metrics that had to be collected to answer the questions. 

Since this methodology consists of multiple specific methods (e.g., 
box structure design, statistical testing, rigorous inspections), each 
particular method had to be analyzed along with the full, integrated, 
Qeanroom methodology in general. As a result of the analysis, 
Qeanroom has been “assessed” as a beneficial approach for the 
SEL (as measured by specific goals of these studies), but specific 
elements of the full methodology had to be tailored to better fit the 
particular SEL environment. The tailoring and modifying resulted 
in a revised Qeanroom process model, written in the form of a 
process handbook [10], for future applications to SEL projects. 


That step is the “packaging” component of the experience factory 
process. 

43. PACKAGING 

The final stage of a complete experience factory is that of pack- 
aging. After beneficial methods and technologies are identified, the 
organization must provide feedback to ensuing projects by cap- 
turing the process in the form of standards, tools, and training. The 
SEL has produced a set of standards for its own use that reflect the 
results of the studies it has conducted. It is apparent that such 
standards must continually evolve to capture modified character- 
istics of the process. (The SEL typically updates its basic standard 
every 5 years.) Examples of standards that have been produced as 
part of the packaging process include: 

• Manager's Handbook for Software Development [11 ) 

• Recommended Approach to Software Development [ 12] 

One additional example of an extensive packaging effort in the 
SEL is a management tool called the Software Management Envi- 
ronment (SME). The concepts of the SME, which is now an opera- 
tional tool used locally in the SEL, have evolved over 8 years. 
This tool accesses SEL project data, models, relationships, lessons 
learned, and managers* rules of thumb to present project charac- 
teristics to the manager of an ongoing project This allows the 
manager to gain insight into the project's consistency with or devi- 
ation from the norm for the environment (Figure 10). 

This example of “packaging” reflects the emphasis that must be 
placed on making results of software projects, in the form of 
lessons learned, refined models, and general understanding, easily 
available to other follow-on development projects in a particular or- 
ganization. 

The tool searches die collection of 15 years of experience archived 
in the SEL to select ap propriate, similar project data so that manag- 
ers can plan, monitor; predict, and better understand their own 
project based on the analyzed history of similar software efforts. 

As an example, all of the error characteristics of the flight dynamics 
projects have resulted in the error model depicted in Figure 8, 
where history has shown typical software error rales in the different 
phases of the life cycle. As new projects are developed and error 
discrepancies are routinely reported and added to the SEL data 
base, die manager can easily compare error rates on his or her proj- 
ect with typical error rates on completed, similar projects. 
Obviously, the data are environment dependent, but the concepts of 
measurement, process improvement, and packaging are applicable 
to all environments. 

5. ADA ANALYSIS 

A more detailed example of one technology that has been studied 
in the SEL within the context of the experience factory is that of 
Ada. By 1985, the SEL had achieved a good understanding of 
how software was developed in the FDD; it had baselined the de- 
velopment process and had established rules, relationships, and 
models that improved the manageability of the process. It had also 
fine-tuned its process by adding and refining techniques within its 
standard methodology. Realizing that Ada and object-oriented 
techniques offered potential for major improvement in the flight 
dynamics environment, the SEL decided to pursue experimentation 
with Ada. 

The first step was to set up expectations and goals against which 
results would be measured. The SEL's well-established baseline 
and set of measures provided an excellent basis for comparison. 
Expectations included a change in the effort distribution of devel- 
opment activities (e.g., increased design and decreased testing); no 
greater cost per new line of code; increased reuse; decreased main- 
tenance costs; and increased reliability (i.e., lower error rates, fewer 
interface errors, and fewer design errors). 
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Figure IQ. SME: A 

The SEL started with a small, controlled experiment in which two 
versions of die same system were developed in parallel: one was 
developed in FORTRAN using the standard SEL structured meth- 
odology, and the other was developed in Ada using an object- 
oriented development (OOD) methodology. Because the Ada 
system would not become operational, analysts had time to investi- 
gate new ideas and leam about the new technology while extracting 
good calibration information for comparing FORTRAN and Ada 
projects, such as size ratios, average component size, error rates, 
and productivity. These data provided a reasonable means for 
planning the next set of Ada projects that, even though they were 
small, would deliver mission support software. 

Over the past 6 years the SEL has completed 10 Ada/OOD 
projects, ranging in size from 38 to 185 KSLOC. As projects com- 
pleted and new ones started, the methodology was continually 
evaluated and refined. Some characteristics of the Ada envi- 
ronment emerged early and have remained rather constant; others 


Tool for ‘Packaging” 

took time to stabilize. For example, Ada projects have shown no 
significant change in effort distribution or in error classification 
when compared with the SEL FORTRAN baseline. However; 
reuse has increased dramatically, as shown in Figure 11. 

Over the 6-year period, die use of Ada and OOD has matured. 
Source code analysis of the Ada systems, grouped chronologically, 
revealed a maturing use of key Ada features, such as generics, 
strong typing, and packaging, whereas other features, such as task- 
ing, were deemed inappropriate for the application. Generics, for 
example, were not only used more often in die recent systems, 
increasing from 8 to 50 percent of the system, but they were also 
used in more sophisticated ways, so that parameterization increased 
eightfold. Moreover, the use of Ada features has stabilized over the 
last 3 years, creating a SEL baseline for Ada development. 

The cost to develop new Ada code has remained higher than the 
cost to develop new FORTRAN code. However, because of the 
high reuse, the cost to deliver an Ada system has significantly 
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decreased and is now well below the cost to deliver an equivalent FORTRAN asse 87/M eaoo fl£V9i 

FORTRAN system (Figure 12). 


Reliability of Ada systems has also improved as the environment 
has matured. Although the error rates for Ada systems, shown in 
Figure 13, were significantly lower from the start than those for 
FORTRAN, they have continued to decrease even further Again, 
the high level of reuse in the later systems is a major contributor to 
this greatly improved reliability. 

During this 6-year period, the SEL went through various levels of 
packaging the AdVOOD methodology. On the earliest project in 
1985, when OOD was still very young in the industry, the SEL 
found it necessary to tailor and package their own General 
Object-Oriented Development (GOOD) methodology [13] for use 
in the flight dynamics environment This document (produced in 
1986) adjusted and extended die industry standard for use in the 
local environment In 1987, the SEL also developed an Ada Style 
Guide [14] that provided coding standards for the local environ- 
ment Commercial Ada training courses, supplemented with lim- 
ited project-specific training, constituted the early training in these 
techniques. The SEL also produced lessons-leamed reports on the 
Ada/OOD experiences, recommending refinements to the method- 
ology. 

Recently, because of the stabilization and apparent benefit to the 
organization, Ada/OOD is being packaged as part of the baseline 
SEL methodology. The standard methodology handbooks [11, 12] 
include Ada and OOD as mainstream methods. In addition, a com- 
plete and highly tailored training program is being developed that 
teaches Ada and OOD as an integrated part of the flight dynamics 
environment. 


COST *TO DEUVER 

EFFORT PER DELIVERED STATEMENT 



FORTRAN 85/86 87/M 88/89 9CV91 


Coat — EMort/Siz# 

Sa> (duvkyd) - New sM s m su ts + 20% ot mussd 
Size (defeated) - Tola) de fe ated statement* 

NOTE: Cort per statement te ueed hate ae the basis lor comparison, since 
the SEL has found s3*»o *1 ratio when comp a ring Ada with 
FORTRAN source Inee of code (carriage returns) but a 1-to-l ratio 
when comparing statements. 


Although Ada/OOD will continue to be refined within the SEL, it 
has progressed through all stages of the experience factory, moving 
from a candidate trial methodology to a fully integrated and pack- 
aged part of the standard methodology. The SEL considers it base- 
lined and ready for further incremental improvement 

6. IMPLICATIONS FOR THE DEVELOPMENT ORGANI- 

ZATION 

For 15 years, NASA has been funding the efforts to carry out 
experiments and studies within the SEL. There have been signifi- 
cant costs and a certain level of overhead associated with these ef- 
forts; a logical question to ask is “Has there been significant bene- 
fit?” The historical information strongly supports a very positive 
answer. Not only has the expenditure of resources been a wise 
investment for the NASA flight dynamics environment, but mem- 
bers of the SEL strongly believe that such efforts should be 


Figure 12. Costs To Develop and Deliver 


commonplace throughout both NASA and the software communit 
in general. The benefits far outweigh the costs. 

Since the SEL's inception in 1976, NASA has spent approximate! 
$14 million dollars (contract support) in the three major strppo 
areas required by this type of study environment: research (defir 
ing studies and analyzing results), technology transfer (producin 
standards and policies), and data processing (collecting forms an 
maintaining data bases). Approximately 50 staff-years of NAS 
personnel effort have been expended on the SEL. During this sair 
period, the flight dynamics area has spent approximately $150 mi 
lion on building operational software, all of which has been part » 
the study process. 
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Figure 13. TVends in Error Rates 

During the pest 15 yean, the SEL has had a significant impact on 
the software being developed in the local environment, and there is 
strong reason to believe that many of die SEL studies have had a 
favorable impact on a domain broader than this ooe environment 
Examples of the changes that have been observed include the fol- 
lowing: 

1. The cost per line of new code has decreased only slightly, 
about lOpexcent — which, at first glance might imply that 
the SELhas failed at improving productivity. Although the 
SEL finds that the cost to produce anew source statement 
is nearly as high as it was 15 years ago, there is appreciable 
improvement in the functionality of the software, as well as 
a tremendous increase in the complexity of the problems 
being addressed [15]. Also, there has been an appreciable 
increase in the reuse of software (code, design, methods, 
test data, etc.), which has driven the overall cost of the 
equivalent functionality down significantly. When die 
SEL merely measures the cost to produce one new source 
statement, the improvement is small; but when it measures 
overall cost and productivity, the improvement is sig- 
nificant. 

2. Reliability of the software haj improved by 35 percent As 
measured by the number of errors per thousand lines of 
code (E/KSLOC), flight dynamics software has improved 
from an average of 8.4 E/KSLOC in the early 1980s to 
approximately 5.3 E/KSLOC today. These figures cover 
the software phases through acceptance testing and deliv- 
ery to operations. Although operations and maintenance 
data are not nearly so extensive as the development data, the 
small amount of data available indicates significant 
improvement in that area as welL 

3. The “manageability" of software has improved dramat- 
ically. In the late 1970s and early 1980s, the environment 
experienced wide variations in productivity, reliability, and 
quality from project to project. Today, however, the SEL 
has excellent models of the process; it has well-defined 
methods; and managers are better able to predict, control, 
and manage the cost and quality of the software being 
produced. This conclusion is substantiated by recent SEL 
data that show a continually improving set of models for 


planning, predicting, and estimating all development 
projects in the flight dynamics environment. There no 
longer is the extreme uncertainty in estimating such 
common parameters as cost, staffing, size, and reliability. 

4. Other measures include the effort put forth in rework (e.g., 
changing and correcting) and in overall software reuse. 
These measures also indicate a significant improvement to 
the software within this one environment 
In addition to the common measures of software (e.g., cost and reli- 
ability), there are many other major benefits derived from a “mea- 
surement" program such as the SEL’s. Not only has the under- 
standing of software significantly improved within the research 
community, but this understanding is apparent throughout the 
entire development community within this environment Not only 
have the researchers benefited, but the developers and managers 
who have been exposed to this effort are much better prepared to 
plan, control, assure, and, in general, develop much higher quality 
systems. One view of this program is that it is a major “training" 
exercise within a large production environment, and the 800 to 
1000 developers and managers who have participated in develop- 
ment efforts studied by the SEL are much better trained and effec- 
tive software engineers. This is due to the extensive training and 
general exposure all developers get from the research efforts contin- 
ually in progress. 

In conclusion, the SEL functions as an operational example of the 
experience factory concept The conceptual model for the SEL 
presented in Section 1 maps to the functional groups discussed 
under SEL operations in Section 3. The experience base in Fig- 
ure 2 is realized by the SEL data base and its archives of man- 
agement models and relationships [16]. The analysis function from 
Figure 2 is performed by the SEL team of software engineering 
analysts, who analyze processes and products to understand the 
environment, then plan and execute experiments to assess and 
rtflni the new technologies under study. Rnally, the synthesis 
function of the experience factory maps to the SEL*s activities in 
packaging new processes and technology in a form tailored spe- 
cifically to the flight dynamics environment. The products of this 
synthesis, or packaging, are die guidelines, standards, and tools the 
SEL produces to infuse its findings bade into the project orga- 
nization. These products are the experience packages of the experi- 
ence factory model. 

Current SEL efforts are focused on addressing two major questions. 
The first is “How long does it take for a new technology to move 
through all the stages of the experience factory?" That is, from 
understanding and baselining the current environment, through 
assessing the impacts of the technology and refining it, to pack- 
aging the process and infusing it into the project organization. 
Preliminary findings from the SEL’s Ada and Cleamoom expe- 
riences indicate a cycle of roughly 6 to 9 years, but further data 
points are needed. The second question the SEL is pursuing is 
“How large an organization can adopt the experience factory mod- 
el?" The SEL is interested in learning what the scaleup issues are 
when the scope of die experience factory is extended beyond a 
single environment. NASA is sponsoring an effort to explore the 
infusion of SEL- like implementations of the experience factory 
concept across the entire Agency. 
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